Analysis of Linsker's Simulations of Hebbian Rules

Linsker has reported the development of centre---surround receptive fields and oriented receptive fields in simulations of a Hebb-type equation in a linear network. The dynamics of the learning rule are analysed in terms of the eigenvectors of the covariance matrix of cell activities. Analytic and computational results for Linsker's covariance matrices, and some general theorems, lead to an explanation of the emergence of centre---surround and certain oriented structures. Linsker [Linsker, 1986, Linsker, 1988] has studied by simulation the evolution of weight vectors under a Hebb-type teacherless learning rule in a feed-forward linear network. The equation for the evolution of the weight vector w of a single neuron, derived by ensemble averaging the Hebbian rule over the statistics of the input patterns, is:! a at Wi = k! + L(Qij + k 2 )wj subject to -Wmax ~ Wi < Wmax (1) j lOur definition of equation I differs from Linsker's by the omission of a factor of liN before the sum term, where N is the number of synapses. Analysis of Linsker's Simulations of Hebbian Rules 695 where Q is the covariance matrix of activities of the inputs to the neuron. The covariance matrix depends on the covariance function, which describes the dependence of the covariance of two input cells' activities on their separation in the input field, and on the location of the synapses, which is determined by a synaptic density function. Linsker used a gaussian synaptic density function. Depending on the covariance function and the two parameters kl and k2' different weight structures emerge. Using a gaussian covariance function (his layer B -+C), Linsker reported the emergence of non-trivial weight structures, ranging from saturated structures through centre-surround structures to bi-Iobed oriented structures. The analysis in this paper examines the properties of equation (1). We concentrate on the gaussian covariances in Linsker's layer B -+C, and give an explanation of the structures reported by Linsker. Several of the results are more general, applying to any covariance matrix Q. Space constrains us to postpone general discussion, and criteria for the emergence of centre-surround weight structures, technical details, and discussion of other model networks, to future publications [MacKay, Miller, 1990]. 1 ANALYSIS IN TERMS OF EIGENVECTORS We write equation (1) as a first order differential equation for the weight vector w: (2) where J is the matrix Jij = 1 Vi, j, and n is the DC vector ni = 1 Vi. This equation is linear, up to the hard limits on Wi. These hard limits define a hypercube in weight space within which the dynamics are confined. We make the following assumption: Assumption 1 The principal features of the dynamics are established before the hard limits are reached. When the hypercube is reached, it captures and preserves the existing weight structure with little subsequent change. The matrix Q+k2J is symmetric, so it has a complete orthonormal set of eigenvectors2 e Ca) with real eigenvalues Aa. The linear dynamics within the hypercube can be characterised in terms of these eigenvectors, each of which represents an independently evolving weight configuration. First, equation (2) has a fixed point at